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Abstract. To establish an evidence base for future computer-assisted language 
learning (CALL) design, CALL research needs to move away from CALL versus 
non-CALL comparisons, and focus on investigating the differential impact of 
individual coding elements, that is, specific features of a technology which might 
have an impact on learning (Pederson, 1987). Furthermore, to help researchers find 
possible explanations for the success or failure of CALL interventions and make 
appropriate adjustments to their design, these studies should be conducted within the 
framework of second language acquisition (SLA) theory (Pederson, 1987). Despite 
this, a recent review found that broad CALL comparisons are still common and 
studies focusing on individual coding elements are rare (Macaro, Handley, & Walter, 
2012). Moreover, few studies make links with SLA and few measure linguistic 
outcomes using measures developed in the field of SLA. One reason for this may be 
difficulty in obtaining the instruments used in SLA research. The IRIS database is 
introduced as one way of addressing this problem. 

Keywords: research methods, instruments, second language acquisition, open 
access. 

1. Introduction 

It is our conviction that more basic CALL research, and replications thereof, are 
required to construct a reliable evidence base upon which future CALL software 
can be designed. Basic CALL research refers to research which allows us to 
develop an understanding of what specific features of digital environments create 
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conditions and engage learners in processes that promote SLA, as well as what task 
variables promote SLA. 

CALL research, however, is largely failing to do this, and what we have instead is 
an accumulation of studies whose findings cannot easily be connected to those of 
studies in the broader field of SLA, or even other studies within CALL itself. Firstly, 
broad atheoretical CALL vs non-CALL comparisons -comparing CALL software 
with ‘pen-and-paper’ or ‘traditional’ classroom activities- are still common in 
the CALL evidence base (Macaro et al., 2012). In such studies, the experimental 
condition tends to differ in multiple ways from the control condition, and as a 
consequence it is not possible to determine to which feature of the software any 
observed differences should be attributed. 

Secondly, most CALL research is not grounded in SLA theory (Macaro et al., 
2012). Grounding CALL research in SLA theory helps researchers to identify 
possible explanations for the effectiveness of particular manipulations of CALL 
environments and make appropriate adjustments to their design to better support 
language acquisition (Pederson, 1987). 

Thirdly, the outcome measures employed in many CALL studies were developed for 
the specific purposes of the study in question and often differ from those commonly 
used in SLA research (Macaro et al., 2012). This is problematic because failure to 
engage in instrumental replication, i.e. to use the outcome measures employed in 
previous research, limits the comparability of studies (Polio, 2012). 

Finally, methods are frequently not adequately reported to permit replication, and 
in particular, instruments are often not provided (Macaro et al., 2012). Replication 
is, however, a cornerstone of scientific enquiry, necessary to ensure the construction 
of a reliable evidence base (Polio, 2012). 

In summary, current approaches to CALL research “are encouraging an 
accumulation of vaguely inter-connected research findings rather than the 
construction of knowledge across independent studies” (Porte, 2013, p. 12) upon 
which future CALL software can be designed. 

In response to this, we introduce some different forms that basic research and 
replication might take in CALL research, and introduce IRIS (www. iris-database, 
org), a digital repository of instruments, materials and stimuli used to elicit data 
in SLA research, as a resource to facilitate replication and promote the design of 
comparable studies. 
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2. Basic CALL research 

As said, we believe that more basic CALL research is required to allow us to 
construct an evidence base upon which to design future CALL. Basic CALL 
research refers to research designed “to discover something about how students 
best learn a language”, which “provid[es] explanatory data and add[s] to the 
theoretical bases for second language learning” (Pederson, 1987, p. 125). It might 
take one of three forms: (1) exploratory research, (2) observational research, or (3) 
narrowly focused experimental research. Exploratory research is characterised by 
ethnographic studies in which researchers observe and interview students about 
their naturalistic use of CALL software with a view of identifying hypotheses 
regarding what features of digital environments create conditions and engage 
learners in processes that promote second language acquisition (Pederson, 1987). 
This might also be achieved through design-based research (see Yutdhana, 2008) 
and educational engineering (see Colpaert, 2006). Observational research refers 
to studies which log the processes that students engage in during software use 
and explore the relationship between software use and learning gains. Narrowly 
focused experimental studies isolate out specific coding elements, i.e. specific 
features of a technology which might have a differential impact on learning, and 
explore hypotheses grounded in SLA theory and research. 

3. Replication in CALL 

Further to more basic CALL research, replication is also required. Exact 
replications, in which researchers attempt to copy the original study as closely as 
possible using identical subjects, conditions, and instruments, among other things, 
should be conducted, where possible, to allow the confirmation of the reliability of 
findings (Polio & Gass, 1997). Instrumental replications, approximate replications 
in which the same outcome measures as used in previous research are employed, 
permit comparisons of findings across studies. Conceptual replications in which 
findings are tested using a different study design, are essential to demonstrate the 
external validity of findings, i.e. “to see if the results hold for a different population, 
in a different setting, or for a different modality” (Polio & Gass, 1997, p. 502), in a 
context in which there is so much individual variation in success. 

Replication in CALL research has, however, largely been neglected, with the 
exception of a number of studies which have replicated findings of SLA research 
(Chun, 2012). Instrumental replication is, however, essential to enable us to connect 
the findings of different CALL studies with one another and studies in the broader 
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field of SLA, and construct an evidence base upon which to base the design of 
future CALL software. 

The problem, however, is that CALL research is not adequately reported to 
permit instrumental replication, let alone exact replication (Macaro et ah, 2012). 
Instruments, including background questionnaires, measures of proficiency, 
instruments for data elicitation and pre- and post-tests, and coding frameworks 
(Polio & Gass, 1997), are rarely provided in CALL studies, and often barely 
discussed in the methods sections of research articles (Macaro et ah, 2012). While 
it is always possible to contact authors to request materials, researchers can be 
difficult to track -they move- and they may not always be able to easily locate 
materials used in their past research (Marsden & King, 2013; Marsden & Mackey, 
2014). 

4. The IRIS database 

Instruments for Research into Second Language Learning and Teaching (IRIS) 
is an open access digital repository of materials used to collect data in research 
on second and foreign language acquisition. All instruments held on the database 
have been used to collect data for a peer-reviewed publication, i.e. a peer-reviewed 
journal or conference proceedings, and edited book, or a successful doctoral thesis. 
The database is searchable along a number of dimensions including instrument 
type, linguistic feature, and learner proficiency, and materials can be downloaded 
and re-used, mostly held under a Creative Commons Attribution Non-Commercial 
Share -Alike licence. 

It is also possible for researchers to upload their own instruments to the database 
for use by other researchers. In fact, 23 top ranking journal editors are now 
encouraging uploads, and IRIS currently holds over 850 documents bundled into 
approximately 280 instruments. The coverage of the database is wide, with over 
fifty instrument types, and over forty research areas represented. 

As a research area, CALL is currently underrepresented with only two instruments, 
in comparison with morphosyntax (grammar) for which there are over 100. In 
line with current interests in computer-mediated task-based language learning, 
however, a variety of tasks are held on the database which might be re-used and 
adapted. These include tasks designed to investigate learners’ use of communication 
strategies (Garcia Mayo, 2005), elicit specific morphosyntactic forms (Mifka 
Profozic, 2012), and examine the impact of task complexity on the extent to which 
focus is on form or meaning (Revesz, 2011). 
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5. Conclusion 

Current approaches to CALL research comparisons have resulted in “an 
accumulation of vaguely inter-connected research findings” (Porte, 2013, p. 12). 
To provide a reliable evidence base upon which to base future CALL design, more 
basic research, and replications thereof, are necessary. Instrumental replication is 
particularly important to permit researchers to build on the findings of previous 
research. To permit such comparisons, CALL researchers are encouraged to 
contribute instruments to the IRIS database. With 4,600 downloads to date and 
references to the publications in which the instruments have been used, having 
materials on IRIS increases the visibility of research. 
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Social Research Council and the British Academy. 
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